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SUMMARY 



Wollmer*s Markov Decision Model for instructional sequence optimiza- 
tion was investigated in a computer-assisted instruction (CAI) context. 
Two special CAI programs served as vehicles for testing the model; one 
program (K-Laws) taught the students to solve DC circuit problems using 
Kirchhoff *s Laws; the other (TRIG) gave practice in manipulating the six 
trigonometric ratios. The K-Laws course had eleven stages or levels; 
TRIG had five; both courses were arranged in a hierarchical order. The 
Wollmer model requires that transfer would occur from one stage to the 
next in a hierarchical learning sequence, and that these effects could 
le estimated so as ^to produce an optimal training schedule. To determine 
the effects of additional practice, half the calibration sample was required 
to finish one .successful trial and half were required to have two successes, 
before moving on to the next stage. Thirty subjects took the K-Laws course, 
80 completed TRIG. Instruction was given at individual CAI terminals. 

All subjects finished the course, and learned to perform satisfac- 
torily the final criterion behaviors. Practice effects were unexpectedly 
slight; people who had one success at each stage of the course had about 
the same criterion-problem performance as those who had two successes 
throughout. The average time required to achieve a second success was 
not appreciably different from that required for the first, and two 
successes at the immediately preceding level was no better than one, as 
far as transfer to the next higher stage was concerned. These results 
indicated that the Wollmer hierarchical model could not improve overall 
learning much by "optimal" scheduling of practice. 

One implication of the findings is that in complex learning hierar- 
chies where the top or most difficult task consists of a collection of 
previously-learned skills, performance time on that top task may be more 
dependent on the number of subskills involved than upon the number of 
practice, trials in preceding stages. Another implication is that if 
practice and transfer effects are to be significant in learning this 
kind of hierarchically-structured material then a very large number of 
practice trials may be necessary. 
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ABSTI^CT 



Two CAI programs in electronics and trigonometry were written to 
test the Wollmer Markov Model for optimizing hierarchical learning; 
calibration samples totalling 110 students completed these programs. 
Since the model postulated that transfer effects would be a function of 
amount of practice, half the students were required to complete one 
practice problem successfully before moving to the next stage; the other 
half had to do two practice problems successfully. 

All students completed the courses satisfactorily. Practice effects 
were small; students who had one success in each stage did about as well 
as those who had two successes. The Wollmer model was thus not suitable 
for optimizing instruction, in terms of minimizing overall time, in these 
particular courses. Perhaps the main reason for this result was that, 
as the student works up to the top of the hierarchy, the sheer number 
of subskills involved in the final task becomes a major determinant of 
performance time, and number of practice trials has a relatively minor 
effect, unless a very large number of practice trials is given. 
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1. INTRODUCTION 

Research at BTL has, for some years, been concerned with different 
aspects of instructional technology required for more effective utilization 
of computers in training and education. Early in this work (Rigney, 1973), 
an outline was prepared of the elements that constitute an instructional 
system. This outline is depicted in Figure 1. Each of the elements shown 
there must be present in some form in a working instructional system. 
The objective of this laboratory has been to allocate its particular cap- 
abilities to research on appropriate elements in this diagram. In those 
instances where the laboratory has produced and field-tested complete 
instructional systems, the best available elements were used in those 
parts of the systems where the laboratory was not, at the time, doing 
research . 

One of the candidates for improvement in instructional systems is 

the "instructioi\al sequence optimizer," which is shown in the adaptive 

controller. Atkinson and his colleagues (Atkinson and Paulson, 1973) 

have convincingly demonstrated the power of certain types of optimization 

models. The interest of this laboratory in this part of the instructional 

system relates to technical subject-matter typical of technical training 

courses in the Navy. It was considered worthwhile to investigate pos- 

« 

sibilities for developing an instructional sequence optimizer based on 
operations research techniques. The initial, Markov Decision Model 
was described by Wollmer (1973). 

Smallwood (1962) was perhaps the first to propose a definite model; 
his optimizer assigned that lesson segment which had the highest utility, 
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or highest expected return in the criterion score. Estimates of utility 
could be made from standardization runs on a sample of subjects. Suppes, 
Atkinson and their colleagues at Stanford have extended optimization of 
instruction into several dimensions; for instance, Atkinson & Paulson 
(.1973) define a model which maps learning performance in terms of three 
aspects: item difficulty, student ability, and learning rate parameters. 
Using this model, optimization of a vocabulary-learning task was accomp- 
lished by estimating the parameters, and giving practice in those items 
which promised the most gain per practice trial. This optimization was 
very successful, yielding an efficiency ga\n on the order of 40 per cent, 
in terms of time saved. Chant and Atkinson (1973) developed an optimiza- 
tion technique for allocating instructional effort to two interrelated 
strands of learning material. Their key assumption was that the learning 
rate for each of the two strands depends solely on the difference between 
the achievement levels on the two strands. 

The Wollmer (1973) model assumes that the course being taught proceeds 
in a definite hierarchical nature. Thus if levels are numbered consecu- 
tively with 1 being the most difficult, and the highest numbered being 
the easiest, mastering of the material at a particular level implies 
mastering of the material at all higher numbered levels. Furthermore, it 
is assumed that successful completion of a problem at one level increases 
the probability of being able to successfully complete a problem at the 
next most difficult level, following instruction at that level. This 
model can be considered a special case of a partially observable Markov 
decision process over an infinite planning horizon. Smallwood and Sondik 
(1973) formulate -and solve such a decision process over a finite planning 
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horizon. The principal purpose of this study was to provide data for 
parameters estimation in this model. 

Research in computational techniques for the more general infinite 
horizon Markov decision process is currently being done at BTL and results 
will be reported in future publications. These results will not only 
offer an alternative computational technique for the model described 
above but also will allow one to relax some of its more restrictive 
assumptions . 
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II. THE TEACHING PROGRAMS 

Two teaching programs were specially written as vehicles for 
testing the Wollmer hierarchical model. One of these programs (K-Laws) 
taught the student to solve DC circuit problems using Kirchhoff *s voltage 
and current laws; the other (TRIG) provided instruction and practice in 
manipulating the six trigonometric ratios- 

The Kirchhoff's Laws Course 

The Kirchhoff*s and Ohm's relations are among the most-taught prin- 
ciples of science. All students in electronics and physics are supposed 
to master them, and of course, many textbooks and courses feature these 
principles throughout. Even so, there is plenty of evidence that simple 
circuit analysis remains difficult for many technicians and students. A 
real problem, apparently, is the designation or translation of physical 
circuit quantities into the Kirchhoff and Ohm equations. Solving DC cir- 
cuit problems via these equations is analogous to working out a "word 
problem*' in alg.-^bra: once the equations are set up, everything can proceed 
smoothly; the difficulty is to translate the verbal statements can con- 
ditions into the algebraic framework. 

The K-Laws course was organized hierarchically, with the desired 
criterion skill at the end of the course being a demonstrated ability to 
calculate certain voltage drops in a three-wire circuit like that shown 
in Fig. 2. For a typical problem near the end of the course, the student 
would be shown the schematic in Fig. 2, with the following parameters: 
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LINE 5 



LINE 6 



Figure 2. Schematic of three-wire 
circuit 
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draws 16 amps 
R2 draws 23 amps 
draws 13 amps 
draws 9 amps 

Generator 1 is delivering 114 volts, polarity as shown 
Generator 2 is delivering 109 volts, polarity as shown 
* Each wire. A, B, C, D, E, F, has a resistance of 0.5 ohms. 

He would then be asked, what is the voltage drop acress R^? 

Such circuit problems cannot b.e solved through guessing. Several 
calculations are necessary, along with careful definition of the relations 
that prevail in the circuit. There are various ways .to find the desired 
answer. For most technicians, an effective method is to determine the 
amount and direction of all the currents, to convert the various current 
loads into voltage drops by multiplying resistances and currents, and 
finally to set up Kirchoff's voltage law in one unknown and solve for 
the missing voltage. 

The requisite skills to accomplish this final criterion performance 
are laid out in Fig. 3, which displays ^ presumed '^learning hierarchy^* 
for the college-level subjects that were used (Gagne, 1970). For this 
sample of people, certain algebraic and verbal skills were assumed; if 
the same criterion skills had to be imparted to seventh-graders, then the 
hierarchy would be considerably extended. 

There are two major paths in the K-Laws learning hierarchy. On the 
left is what might be called the "voltage drop" sequence; here the student 
learns or reviews the Ohm's law formulas and practices using them in 
several circuits; he also applies- the "sign rule" regarding the direction 
of current flow and the sign of the voltage drop through a resistance. 
At the right side of the hierarchy, there is a chain of subskills involving 
the determination of current direction and quantity in a three-wire, two 
generator circuit with several passive loads. Here the information to be 
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Find Voltage Drops in any of the loads in 
DC circuit shown in Figure 2, 



Eri=ei- Eli- El3 



Er3= E2- El3- El5 
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Solve Kirchhoff's 


Voltage 


& Current Laws in 


one 


Unknown 





Isolate smallest closed 
loop containing unknown load 
and an active voltage source 
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Use *'sign rule" to 
mark resistances as 
"additive" or "sub- 
tractive" voltage 
drops 



Mark direction of 
current in Line 3 
l3=left if I^>l5 
l3=right if I^<l5 
13=0 if I3-I5 



Calculate voltage 
drops in loads & 
lines according to 
Ohm*s Law 



Mark direction of 
current in Line 4 
l4=left if I2>l6 
I.=right if I.<I. 
14=0 if 12=15' 
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in line 4= 
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Mark current in 
line 1 = current 
drawn by load 1 




Mark current in 
line 2 = total 
current drawn 
by loads 1 & 2 




Mark current in 
line 4 = total 
current drawn 
by loads 3 & 4 




Mark current in 
line 3=current 
drawn by load 4 
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i 
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Mark direction of current in out- 
side wires (right in lines 1 & 2, 
left in lines 5 & 6) 



Establish negative cui^rent flow con- 
vention (electrons move away from neg 
ative pole> toward positive pole) 



T 



Designate Unknown Load 




Figure 3 - A Gagne' hierarchy for the K-Laws problem solving procedures 
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taught is, perhaps, less general than the voltage-drop material; the 
solution sequence is confined to a particular three-wire configuration, 
and might not hold exactly for similar but different circuit layouts. 
The voltage drop principles, in contrast, are extremely broad in applica- 
tion. Both chains of the hierarchy have to be mastered to solve the 
final criterion problem. 

If a learning hierarchy is valid, then substantial and positive 
transfer from "lower" to "higher" stages should obtain; that is, those 
who can perform the subskills "underneath" a final behavior should be 
more likely to succeed in the final task, than those who cannrt accomplish 
the subskills. Gagne and his associates (1962) demonstrated such transfer 
effects in a mathematics task with school children. For the K-Laws pro- 
gram, it was assumed that it would be a good instructional strategy to 
require every subject to demonstrate a definite capability at every sub- 
skill level, before advancing to the next part of the course. 

Eleven "levels" or course stages were defined; those units corres- 
ponded roughly to tasks in the learning hierarchy. Levels were numbered 
so that high numbers represents easy or early parts of the course. Some 
levels were very elementary and easy, such as the lesson involving direction 
of electron flow out of a battery. The last two or three stages, though, 
were quite involved, since the student was then applying several newly- 
learned skills from all or most of the preceding stages, and he was usually 
performing these skills in a definite order. 

To the student, a standard teletype terminal was the major piece of 
teaching hardware in the set up. This terminal was, of course, driven by 
a time-shared teaching program, which was written in the BASIC language 

20 
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and was stored in a distant central computer, A random access slide pro- 
jector was used to display circuit diagrams on the wall, in front of the 
student. Two booklets vere also part of the teaching package; one contained 
lesson material and illustrations, the other had blank circuit diagrams 
for the student to use as he practiced some of the lessons. A small hand- 
held calculator was provided for calculations; each student worked a few 
problems on it before beginning the course. 

When a student appeared for instruction and first logged on to the 
teletype, the program asked him eleven questions regarding his knowledge 
of electronics. These questions related directly to the eleven stages of 
K-Laws course content. In fact, question number 11 essentially asked the 
student if he knew level 11, question 10 if he could perform the criterion 
tasks in level 10, etc. If he answered "yes" to a question, then he was 
given a sample problem to determine if that yes answer was valid. For 
example, at level 6, question 6 was: Can you calculate the resistances 
in a parallel DC circuit?" If a student typed a "yes" to this question, 
then a single test item was given to him, to see if he actually could per- 
form. The program generated a circuit of three or four resistors in parallel, 
and asked the student to figure out the total resistance across them. Com- 
parison of the student's answer to the correct one was immediately performed 
by the program and printed out for the student to see. The entering skills 
test yielded, then, a series of eleven "yes" or "no" answers, along with 
a pre-test right-wrong score for each "yes" item. 

Once the student began the main K-Laws course, he worked through 
the program at his own pace, with a research staff member standing nearby 
to handle such matters as computer shutdowns, log -outs, restarts, and timer 
resets. The staff member did not supplement the lesson materials or attempt 
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to explain difficult items ♦ When learning difficulties did occur,, and 
the student was -perplexed, he was told to go back over the lesson material 
carefully, in step-by-step fashion* When a subject finished studying the 
material in a given teaching unit, he typed "D** (for "Done") on the key- 
board* The program then made up practice problems for that teaching unit, 
with some remediation loops automatically keyed to errors* 

Generally, the system worked satisfactorily. On one or two occasions, 
the problem-generator in the master program happened to produce degenerate 
or "insoluble" problems for the student. Some of the data were obtained 
via time-sharing with a computer just half a mile across campus; the rest 
of the data came by operating through a time-sharing center some 400 miles 
away from the teaching terminal. For this distant operation, noisy tele- 
phone lines caused total shutdown a few times. As it turned out, a few 
students elected not to use the slide projector to display the circuit 
diagrams; they referred solely to the booklets for that information. 

After each problem answer was received by the computer terminal, the 
system immediately printed out a "correct" or "incorrect" evaluation of 
the answer provided by the student, furnished the correct answer, and 
indicated the time that the student had spent on that problem. If an 
answer was wrong, the student had to keep working in that same level, until 
either one correct or two correct answers were achieved. Whether the 
student received one practice problem or two problems was decided by a 
coin flip, when he logged onto the system. 

Thirty subjects completed the K-Laws course. Fifteen of them were 
college students; fifteen were military technicians who were working in 
electronics or related fields at McClellan Air Force Base, California. 
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College students were paid a nominal hourly fee for participating; the 
military people were ordered to appear, and their only recompense was 
time off from regular duties. The subjects differed markedly in their 
familiarity with electronics concepts. Several of the college subjects 
were engineering majors and had completed one or more electricity courses; 
such students might claim that they already knew much of the material in 
the course, but no student could solve the pretest problems in the last 
three (most difficult) stages without some practice at the terminal. At 
the other extreme were some liberal-arts majors who had almost no tech- 
nical experience with voltage drops and circuit diagrams; some of these 
subjects said that they "weren't very good at this sort of thing," but 
all of them persisted and solved the criterion problems at the end. 
Breaks were given about every one-and-a-half to two hours during the 
teaching. In most cases the program was completed in a single day of 
six to eight hours; about a third of the subjects had to appear on two or 
more days because of personal scheduling difficulties, system breakdowns, 
and the like. 

The Trigonometry Course 

A short course in trigonometry (TRIG) served as the second vehicle 
to test the model. The TRIG course consisted of five levels, and as in 
the K-Laws program, they were arranged in a strict hierarchical structure. 
The levels were numbered so that level five indicated the easiest or enter- 
ing lesson, and level one represented the most difficult. In order for 
a student to know the material at a given level (say Level 2), he also had 
to use significant parts of the material at all higher numbered levels 
(say Levels 5 through 3). 
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As a start) in Level 5 > the student was given the definition of 
the six basic trigonometric ratios — the sine, cosine, tangent, cotangent, 
secant, and cosecant. Then he was presented with a right triangle which 
had the side lengths displayed, and was asked to find the six basic ratios for 
that triangle. Level 4 treated the cof unction relationships; in this unit 
the student learned that the cosine, cotangent, and cosecant of an angle 
are equal to the sine, tangent, and secant of the complementary angle. 
Level 3 instruction used the relation Sin 0 + Cos 9 = 1, and gave practice 
in working out values from this equation. Level 2 taught how all the trig 
ratios can be computed from either the sine, cosine, secant, or cosecant. 
Finally, in Level 1 , the most advanced unit, the student was shown how to 
determine all six ratios from either the tangent or cotangent. Then he 
was given either the tangent or cotangent of an angle, and asked to find 
the other five basic trigonometric ratios. Satisfactory performance in 
this last teaching unit resulted in "graduation." 

As Ln the K-Laws sequence, the student was asked questions about 
the material in a pretest session, before he began the instruction. Thus, 
one question, keyed to Level 3, asked: "Do you know how to compute the 
cosine of an angle from its sine?" There were five such preliminary 
questions, one for each level. 

After the student answered these five questions he was given instruc- 
tions and problems at levels five, four, three, two and one in that order. 
A student advanced from one level to another by successfully solving 
either one or two problems at that level, the number for each man being 
determined by a random number generator. In order for a student to gain 
credit for a problem, he had to do all parts correctly. Thus if a student 
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received a problem at level five and gave an incorrect answer for one of 
the six trigonometric ratios, he was immediately informed that he missed 
that problem and then presented with a new triangle, and was asked to 
solve for a new series of six ratios. Before being given a new problem 
the student always had the option of reviewing instruction. 

Eighty TRIG subjects were run on a PLATO IV terminal. Subjects 
were psychology students who received "subject pool" course credit for parti- 
cipating; each one was scheduled for two hours at the terminal, but most 
did not require that long to finish the course. TRIG was written in the 
TUTOR language. 



III. PARAMETER ESTIMATION 

Two probability vectors are needed by the Wollmer (1973) model. The 
components of these vectors are: 



p. = p 



student can initially solve a problem successfully at 
level i 



1 



student can perform at level i-1 / student solves a problem 
correctly at level i and could not perform successfully at 
level i before . 



A large group of subjects were run, as described in Section II, to 
collect data that could be used for estimating initial values of these 
parameters . 

According to the model, the probability that a student can perform 

successfully at level i after solving k problems successfully at level 

k — 

i+1 is l-(l-Pj^) (^'•Qi+i) • X^^ be the proportion of incorrect solu- 

tions at level i by students who solved one problem correctly at level i+1, 
and let be the proportion of incorrect solutions at level i by students 
who solved two problems correctly at level i+1. Then X^^ and 
estimators of quantities as follows: 

This is, X^. and estimate the probability of failure at level i by 
students who solved one and two problems correctly respectively at level 
i+1. Solving these for p^ and q^^^j^ one obtains: 

■Pi = 1 - 4/^2i . 
= 1 - ^2i/^li 
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The quantities Xj^. and X2j^ may depend on the student's answers to 
the pretest questions. Thus in a five-level teaching system such as TRIG, 
if p£ and q^^j^ are to be estimated solely on the basis of the answer to 
pretest question j, one would obtain X^^ = Y^J(y^^. + Y. .) and 

= •46'^^^ij'*^ ^ ^ij+6^ student answered yes to pretest questior 

= Z. ./(X. . + Z. .) and = Z. (X. + Z. if the student 

answered no to pretest question j. (Full details, definitions of W, X, Y> 
and Z, and data matrices are given in Appendix 1.) 

Since the student answered either yes or no to five (or eleven) 
different pretest questions, there were five (or eleven) possible esti- 
mates of X^j^ and and consequently of p^ and q^^^- Each of these 
estimates for X^^ and X^^ were obtained from the control group and are 
displayed in Tables 9 through 12., There are several ways of estimating 
the p and q vectors from this data, if the model is used to guide students 
through the course. For example, for TRIG (K-Laws) one might average 
estimates of X^j^ and X2j^ and use this to solve for p-^ and '^^^i in (3) and 
(4). Another method is to obtain five possible estimates of and 
^i+1 substituting the pairs of estimates of X^j^ and X2^y and then to 
average these. A third possibility is to base the estimates of p-^ 
and ^^^i solely on the basis of the answer to i which is the one 
specifically directed at level i. Still another way is to let a yes 
answer to a question be considered a one score, to count a no answer as 
zero score, and then let and q^^-^ be a linear combination of the scores 
on the pretest questions. 

First, however, it was necessary to run initial samples through the 
program to provide "calibration" data. 
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Two samples of students were run through all levels of each course, 
under either a two, or one success per-level policy. Allocation of policies 
was by random assignment to students at entry in the K~Laws course, and 
by random assignment to students at each successive level in the TRIG 
course. Answers to pretest questions were tabulated, but were not used 
for weights nor for entrv -level decisions (all students took all levels), 
in these initial, "calibration" samples. 
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IV. RESULTS: CALIBRATION DATA 



The questions of central importance to evaluating the usefulness 
of this type of model are (I) whether the policy of requiring two successes 
at a level resulted in better performance at the next level than the one- 
success-at-a-level policy, and (2) whether learning occurred within a 
level, as indicated by comparing first success with second success data. 
The data in Tables 1 and 2, from the K-Laws course, bear on these questions. 



Table 1 

Overall Failure Rates: (1) For One Success Within a 
Level and (2) for Two Successes 
Within a' Level; K-Laws Course 



One Success 


Policy (N = 15) 


Two Success 


Policy (N = 15) 


Levels 


1 


1st 


2nd 


11 


.079 


.071 


.071 


10 


.079 


.133 


.000 


9 


.143 


.071 


.133 


8 


.133 


.278 


.235 


7 


.278 


.294 


.250 


6 


.235 


.235 


.133 


5 


.435 


.519 


.278 


4 


.308 


.315 


.167 


3 


.400 


.593 


.389 


2 


.538 


.091 


.333 


1 


.500 


.556 


.385 


Mean 


.284 


.287 


.216 


SD 


.166 


.195 


.127 



t (within levels) = 1.716 p(lOdf) = .12 



t (between levels) = .033 p(20df) = .97 
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Although there was a slight reduction in probability o£ failure for 
the second success within a level, from .28 to .22, this was not a statis- 
tically significant difference. This suggests that the policy of requiring 
two successes per level was too "lenient;" that is, not enough extra 
practice was req»)ired to differentiate between policies. Comparison of 
failure rate means for the first success at a level, column 1, also reveals 
practically no difference (.284 and .287) between the effects of the two 
policies. Thus, requiring students to succeed twice within a level did 
not reduce their failure rate for the first success at the next level. 
This overall failure rate was the same as that for students who vjere re- 
quired to succeed only once at a level. 

Examination of the time data for the same conditions (Table 2), 
reveals a similar story. 



Table 2 



Mean Time (Minutes) per Problem: (1) for 
One Success within a Level and (2) for 
Two Successes within a Level; K-Laws Course 



One Success Policy 
1 



Two Success Policy 



Levels 



1st 



2nd 



! 8 
t 7 



! 3 
I 2 
' 1 



11 
9 



5 
4 



.80 
.88 
.93 
3.53 
3.76 
1,73 
8.18 
8.45 
7.66 
4.26 
9.88 



.97 
.86 

.97 
4.00 
2.16 
3.60 
7.42 
6.87 
7.58 
4.61 
7.68 



1.0 
1.93 
1.25 
2.63 
2,33 
2.77 
5.83 
■'6';07 
4.88 
4.83 
8.23 



1 Mean 
SD 



4.55 4.25 
3.43 2.79 



3.80 
2.31 



t (within levels) = 1.347 p (l\Odf) = .21 
t (between levels) = .228 p (20df) = .82 
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In Table 2, comparison between the overall means (4.55 vs 4.25) 
for the first success within a level, for tha two policies, obviously 
yielded no significant difference. While there was a slight overall 
decrease in mean time to achieve the second success within a level in 
comparison to mean time to achieve the first, this difference was neither 
statistically nor practically significant. Further, the two-success policy 
did not have a cumulative effect between levels. Otherwise, it should 
have resulted in a smaller overall mean time to achieve the first success 
in a level than did the one-success policy. Other data from the calibra- 
tion samples are summarized in "Appendix 1. 
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V. DISCUSSION OF RESULTS. 

It is clear from the comparisons in Tables 1 and 2 that different 
amounts of practice on the same problems (within a course level) had only 
slight effects on probability of failure on subsequent problems of the 
same kind, and did not positively influence performance at the next level. 
Since the Wollmer Markov decision model requires that this positive in- 
fluence occur, the model apparently cannot be applied to the kind of 
course material that was used here. It also is clear that the extra 
practice the student received under the two-success policy did not have 
an appreciable effect on mean time to successful solution of a problem 
at the next level. Again, the model i^equires that the effects of practice 
transfer across levels. The problems used in the K-Laws and TRIG programs 
were relatively complicated, in that each problem consisted of several 
parts, and required that the students perform a series of operations, 
often in a certain sequence. Under these circumstances, time to perform 
should be determined by the number of operations to be performed, until 
the practice-f or-f luency stage is reached, at which "chunking" of operations 
can occur; this might reduce the correlation between time to perform and 
number of operations to be performed. If this phenomenon occurs, it is 
likely to occur only after long periods of ovcrlearning, entailing in- 
tensive practice, indeed. 

In a radar-intercept trainer (Rigney, et al^. , 1974), students 
performed the same six mental arithmetic problems^ over and over. Over a 
series of 100 to 150 practice problems, in 10 to 20 sessions, involving 
10 to 15 hours of practice, overall mean latency to do all six problems was 
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reduced from 68 seconds to 29 seconds, or, by a factor of 2.3. It should 
be noted also that this was a real-time situation, in which students were 
driven to perform faster by the requirement to keep up with a developing 
tactical problem. The present study did not impose this degree of time 
pressure on the learner. 

-The different levels in the K-Laws and TRIG courses were created by 
introducing new rules, or procedures. While the student needed to use 
virtually all of the rules or procedures he had learned at preceding levels, 
it still seems likely that the new elements in problems at each succeeding 
level were sufficiently novel to reduce inter-level transfer effects. 

To see if a much larger amount of practice would affect time to 
perform, three additional K-Laws ^subjects were run under the condition that 
five correct solutions were required at each of the eleven levels before 
"graduation" from a stage of instruction. This policy did seem to promote 
more learning and transfer; the mean time for problem solution in the 
final stage was 5.3 minutes, compared to 8 or 9 minutes for the one-success 
and two-success policies. But a much longer total training time, on the 
order of hours, was required to get this two or three minute improvement 
at the final level. Because of this relative inefficiency, the Wollmer 
model would not prescribe >such extensive additional practice. 
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APPENDIX 1 

Tables 3 through 8 present the basic success-failure data from TRIG 
and K-Laws subjects. These data were cross-tabulated so as to indicate the 
correlation between pretest-question responses and learning performance. 
As an example, from Table 5, it appears that there were seven persons who 
answered "Yes" to pretest question 6 and succeeded in doing their level-6 
learning without a failure; Table 7 reveals that there were only two 
persons who answered "Yes" to question 6, and then failed at least once 
when they actually attempted the level 6 material. As a rough estimate, 
then, the probability is something like 7/9 that, if a person answers "Yes" 
to pretest item 6, he will go through the level 6 teaching without a mis- 
take. Wollmer's model and estimation procedures are designed to take 
advantage of such contingencies. 

Estimate of X,. and X^ . are given in Tables 9 and 10 for the TRIG 
li 2i ^ 

instruction and in Tables 11 and 12 for the Kirchhoff^s Laws course. 

Tables 13 and 14 show the mean times on time spent at each level by 
students. For the K-Laws course, the computer system recorded the time 
in minutes with 1 second for less than one minute, 61 seconds for between 
one and two minutes, and so forth. To compensate for this recording 
circumstance, 29 seconds were added to all times shown in Table 14. 

Note that Tables 9 through 12 give the proportion of failures for 
TRIG and K-Laws based on the number of successes at the preceding level. 
However, at level 5 for TRIG (11 for K-Laws) there is no preceding level. 
For these levels, all data are grouped under the 1 success tables. Thus 
Table 10 (12) has no entry for level f (11). 
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Table 5 

The Number of Successes by K-Laws Students 
Who Answered Yes to the Pretest Questions 
(W Matrix) 



LEVEL (i) 

5 6 7 8 9 10 11 

11 222333333 35 
10 444555555 5 11 

+ 

93 3 3 4 4 4 4 4 4 4.6.^ 

8333444444 48^ 

(1) 

7333333333 3 11 ^ 

rt 

6 6 6 6 7.7 7 7 7 7 7 21 § 

4J 

5 9 9 10 8 10 10 10 10 9 9 28 ^ 

o 
(fl 

466767777 7 6 25 ij 

o 

3667677777 6 19 ^^ 

o 
o 

2 10 11 12 9 9 12 12 12 11 11 33 
1 11 12 13 10 13 13 13 13 12 11 33 



G 
O 

4J 
(fl 
(U 

c/lloO 0 0 0 0 0 0 0 0 

S 10 444444444 40 
<u 4- 

;^^90 0 0 0 0 0 0 0 0 0 0"'^ 

8000000000 oo| 



7666666666 6 
6 12 10 12 10 12 12 12 12 12 12 



4J 

(fl 
C 

o 



4J 



5 16 14 16 14 16 16 16 16 16 16 0 ^ 

o 

4 16 14 16 14 16 16 16 16 16 16 0 

o 

3 10 10 10 10 10 10 10 10 10 10 0 

o 
o 

20 18 20 18 20 20 20 20 20 20 
18 16 18 16 18 18 18 18 18 18 
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Table 6 



The number of Successes by K-Laws Students 
Who Answered No to the Pretest Questions 
(X Matrix) 















LEVEL (i) 


















). 


2 
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Table 7 



The Number of Failures by K-Laws Students 
Who Answered Yes to the Pretest Questions 
(Y Matrix) 
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T^ble 8 



The Number of Failures by K-Law Students 
Who Answered No to the Pretest Questions 
(Z Matrix) 
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Table 9 



Proportion of Unsuccessful Attempts in TRIG Course 
Following 1 Correct Solution at the Previous Level 
(Question j Corresponds to Level i) 









PRETEST QUESTION NUMBER (j) 










1 


2 


3 


4 


5 






5 


.429 


• 394 


• 430 


.341 


.347 






4 


. 125 


. 0 


• 038 


• 0 


.083 


r to 
stior 




3 


.375 


.412 


.361 


•353 


.313 


01 01 




2 


.429 


.395 


•357 


.435 


.375 


V) 
W G) 




1 


.450 


• 682 
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• 333 


.400 


4J 

u 


•r-l 
















Level 


5 


.420 


.433 


.418 


.440 


.455 


c 




4 
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.333 


.350 


.309 


.371 


0 

0 -rl 
4J 4J 
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.432 
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V) 

u <u 
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.591 
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Pretest ( 
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Table 10 



Proportion of Unsuccessful Attempts in TRIG Course 
Following 2 Correct Solutions at the Previous Level 
(Question j Corresponds to Level i) 









PRETEST QUESTION NUMBER (i) 
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.781 


.667 


.667 


Yes 
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Table 11 



Proportion o£ Unsuccessful Attempts in Kirchhoff Laws Course 
Following 1 Correct. Solution at the Previous Level 
(In this Table, Question j is Directed to Level 12-i) 
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10 
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PRETEST QUESTION NUMBER (j) 
8 7 6 5 4 
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2 
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1 


.333 
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.250 


.143 


.100 


0 
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2 
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0 


0 


0 


0 


0 


0 


0 


0 


0 
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0 


0 
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c 
o 
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.286 


.200 


.333 


.250 


.222 


.200 


.143 


.250 
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5 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


.236 


est 




6 


0 


0 
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.200 


.250 


.222 


.091 


.125 
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.400 


.286 


.333 


.333 


;.400 


.222 


.333 


.417 


.417 
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0 
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0 


0 


0 
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.125 
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t« 
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>^ 


■r-l 
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0 
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0 


0 
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0 


0 


0 


.029 
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1 


.389 


.438 


.444 


.412 


.412 


.500 


.636 


.533 


.500 


.636 


0 






2 


.214 
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.273 
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.300 


.429 
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.100 
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.143 
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0 
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.267 
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.333 ' 
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0 
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.083 


.083 


.077 


0 
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0 


0 


0 


1 pre 
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.333 
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.111 


.111 
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0 


o 
u 
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.250 
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.250 


.333 
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.270 


.273 
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0 
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.100 


.091 


.091 


.083 


.125 
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.125 


.125 


.250 


0 


TO 

o 
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10 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 
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0 
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0 
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0 


0 


0 


0 


0 


0 





44 

-33- 

ERIC 



Table 12 



Proportion of Unsuccessful Attempts in Kirchhoff Laws Course 
Following 2 Correct Solutions at the Previous Level 
(In this Table, Question j is directed to Level 12ri) 
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.231 
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0 
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\j 
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.294 
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0 
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0 
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.077 
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0 


0 


0 


0 


0 


pret 
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.200 
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4J 
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.294 


.364 
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.167 
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.125 
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Table 13 



Time Data for the TRIG Course (N = 80) 



1 

Level 


Total Time (Sees) 


Number of 
Problems 


Average Time (Min.) 
per Problem 


1 


/Do jU 


T 1 A 


t . Uj 


2 


81788 


251 


5.43 


3 


82599 


262 


5.25 


4 


19620 


199 


1.65 


5 


29015 


237 


2.03 



Table 14 

Time Data for the K-Laws Course (N = 30) 



1 

Level 


Total Time (Sees) 


Number of 
Problems 


Average Time (Min.) 
per Problem 


1 


40771 


72 


9.91 


2 


16754 


62 


4.98 


3 


25774 


71 


6.53 


4 


21755 


62 


6.33 


5 


29483 


76 


6.95 


6 


8455 


54 


3.10 


! ' 


6870 


60 


2.40 


: 8 


9031 


61 


2.95 


9 


1728 


48 


1.08 


10 


1185 


45 


.92 


11 


1244 


44 


.95 
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ATTN: Dr. J. H. Steinemann (3) 
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Navy Personnel Research and 

Development Center 
San Diego, CA 92152 
ATTN: Library (All) 

Navy Personnel Research and 

Development Center 
Code 9041 

San Diego, CA 92152 ^ (13) 

ATTN: Dr. J. D. Fletcher 

D. Gragg, CAPT, MC, USN 
Head, Educational Programs 

Development Department 
Naval Health Sciences Education 

and Training Command 
Bethesda, MD 20014 (134) 



Army 



Technical Director 

U.S. Army Research Institute for 

the Behavioral & Social Sciences 
1300 Wilson Blvd. 

Arlington, VA 22209 (All) 



Dr. Ralph Dusek 

U.S. Army Research Institute for the 

Behavioral and Social Sciences 
3100 Wilson Blvd. 

Arlington, VA 22209 (All) 
Dr . Joseph Ward 

U.S. Army Research Institute for the 

Behavioral and Social Sciences 
1300 Wilson Blvd. 

Arlington, VA 22209 (All) 



HQ USAREUR & 7th Army 
ODCSOPS 

USAREUR Director of GED 
APO New York 09403 



ARI Field Unit - Leavenworth 

Post Office Box 3122 

Fort Leavenworth, KS 66027 



(1234) 



(AH) 



Mr. James Baker 

U.S. Army Research Institute for the 

Behavioral and Social Sciences 
1300 Wilson Blvd. 

Arlington, VA 22209 (3) 



Armed Forces Staff College 
Norfolk, VA 23511 
ATTN: Library 

Commandant 

U.S. Army Infantry School 
Fort Benning, GA 31905 
ATTN: ATSH-DET 



(1234) Air Force 



(123) 



Deputy Commander 

U.S. Army Institute of Administration 
Fort Benjamin Harrison, IN 46216 
ATTN: EA (123) 

Dr. Frank J. Harris 

U.S. Army Research Institute for the 

Behavioral and Social Scienses 
1300 Wilson Blvd. 
Arlington, VA 22209 (3) 

Dr. Stanley L. Cohen 

U.S. Army Research Institute for the 

Behavioral and Social Sciences 
1300 Wilson Blvd. 

Arlington, VA 22209 (123) 



1 Research Branch 
AF/DPMYAR 

Randolph AFB, TX 78148 (1234) 

1 Dr. G. A. Eckstrand (AFHRL/AST) 
Wright-Patterson AFB 

Ohio 45433 (1234) 

1 Dr. Ross L. Morgan (AFHRL/ASR) 
Wright-Patterson AFB 
Ohio 45433 (34) 

1 AFHRL/DOJN 
Stop #63 

Lackland AFB, TX 78236 (1234) 

1 Dr. Martin Rockway (AFHRL/TT) 
Lowry AFB 

Colorado 80239 (All) 
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1 Major P. J. DeLeo 

Instructional Technology Branch 

AF Human Resources Laboratory 

Lowry AFB, Co 80230 (123) 

1 Dr., Alfred R. Fregly 
AFOSR/NL 

1400 Wilson Blvd. 

Arlington, VA 22209 (1234) 

1 Dr. Sylvia R. Mayer (MCIT) 
Headquarters 

Electronic Systems Division 
LG Hanscom Field 

Bedford, MA 01730 (234) 

1 Capt. Jock Thorpe, USAF 
Flying Training Division 
AFHRL/FT 

Williams AFB, AZ 85224 (3) 

1 AFHRL/PED 
Stop 3 

Lackland AFB, TX 78236 (1234) 



Marine Corps 

1 Director, 

Office of Manpower Utilization 
Headquarters, Marine Corps 
Code MPU) 

MCB (Building 2009) 

Quantico, VA 22134 (1234) 

1 Dr. A. L. Slafkosky 

Scientific Advisor (Code RD-1) 
Headquarters, U.S. Marine Corps 
Washington, D.C. 20380 



Coast Guard 

1 Mr. Joseph J. Cowan, Chief 
Psychological Research Branch 
(G-P-1/62) 

U.S. Coast Guard Headquarters 
Washington, D.C. 20590 (All) 



Other DOD 

1 Military Assistant for Human 
Resources 
Office of the Secretary of Defense 
Room 3D129, Pentagon 

Washington, D.C. 20301 (AH) 

1 Advanced Research Projects Agency 
Administrative Services 
1400 Wilson Blvd. 
Arlington, VA 22209 

ATTN: Ardella HoUoway (1234) 

1 Dr. Harold F. O'Neil, Jr. 

Advanced Research Projects Agency 
Human Resources Research Office 
1400 Wilson Blvd. 

Arlington, VA 22209 (134) 

1 Dr. Robert Young 

Advanced Research Projects Agency 
Human Resourses Research Office 
1400 Wilson Blvd. 

Arlington, VA 22209 (1234) 



12 Defense Documengation Center 
Cameron Station, Building 5 
Alexandria, VA 22314 
ATTN : TC 



(A 11)^ 



Chief, Academic Department 

Education Center 

Marine Corps Development and 

Education Command 
Marine Corps Base 
Quantico, VA 22134 



(13) 



Other Government 

1 Dr. William Gorham, Director 

Personnel Research and Development 
Center 

U.S. Civil Service Commission 
1900 E. Street, N.W. 
Washington, D.C. 20415 
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1 Dr. Vern Urry 

Personnel Research and Development 
Center 

U.S. Civil Service Commission 
xl900 E Street, N.W. 

Washington, D.C. 20415 (1234) 

1 Dr. Erik McWilliams, Director 
Technological Innovations in 

Education Group 
National Science Foundation 
1800 G Street, N.W. , Room W 650 
Washington, D.C. 20550 (3) 

1 Dr. Richard C. Atkinson 
Deputy Director 
National Science Foundation 
1800 G Street, N.W. 

Washington, D.C. 20550 (3) 

1 Dr. Andrew R. Molnar 

Technological Innovations in 

Education Group 
National Science Foundation 
1800 G Street, N.W. 

Washington, D.C. 20550 (13) 

l"" Dr. Marshall S. Smith 
Assistant Acting Director 
Program on Essential Skills 
National Institute of Education 
Brown Bbilding, Room 815 
19th and M Streets, N.W. 
Washington, D.C. 20208 (3) 



Miscellaneous 

1 Dr. Scarvia B. Anderson 
Educational Testing Service 
17 Executive Park Drive, N,E. 
Atlanta, GA 30329 (13) 

1 Dr. John Annett 

Department of Psychology 
The University of Warwick 
Coventry CV47AL 
ENGLAND 



1 Mr. Samuel Ball 

Educational Testing Service 
Princeton, NJ 08540 (13) 

1 Dr. Gerald V. Barrett 

University of Akron 

Department of Psychology 

Akron, OH 44325 (1234) 

1 Dr. Bernard M. Bass 
University of Rochester 
Graduate School of Management 
Rochester, NY 14627 (123) 

1 Dr. Ronald P. Carver 
School of Education 
University of Missouri-Kansas City 
5100 Rockhill Road 
Kansas City, MO 64110 

1 Century Research Corporation 
4113 Lee Highway 

Arlington, VA 22207 (1234) 

1 Dr. Kennath E. Clark 
University of Rochester 
College of Arts and Sciences 
River Campus Station 
Rochester, NY 14627 (1234) 

1 Dr. Allan M. Collins 

Bolt Beranek and Newman, Inc. 

50 Moulton Street 

Cambridge, MA 02138 (3) 

1 Dr. Rene* V, Dawis 

University of Minnesota 
Department of Psychology 
Minneapolis, MN 55455 (123) 

1 Dr. Ruth Day 
Yale University 
Department of Psychology 
2 Hillhouse Avenue 
New Haven, CT 06520 (3) 
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1 ERIC 

Processing and Reference Facility 
4833 Rugby Avenue 

Bethesda, MD 20014 (All) 

1 Dr. Victor Fields 
Montgomery College 
Department of Psychology 
Rockville, MD 20850 (All) 

1 Dr. Edwin A., Fleishman 
Visiting Professor 
University of California 
Graduate School of Administration 
Irvine, CA 92664 (1234) 

1 Dr. Robert Glaser, Co-Director 
University of Pittsburgh 
3939 O'Hara Street 
Pittsburgh, PA 15213 (13) 

1 Dr. Henry J. Hamburger 
University of California 
School of Social Sciences 
Irvine, CA 92664 (3) 

1 Dr. M. D. Havron 

Human Sciences Research, Inc. 

7710 Old Spring House Road 

West Gate Industrial Park 

McLean, VA 22101 (1234) 

1 HumRRO Central Division 
400 Plaza Building 
Pace Boulevard at Fairfield Drive 
Pensacola, FL 32505 (123) 

1 HumRRO /We stern Division 
27857 Berwick Drive 
Carmel, CA 93921 

ATTN: Library (1234) 

1 



1 HumRRO/Western Division 
27857 Berwick Drive 
Carmel, CA 93921 

ATTN: , Dr. Robert Vineberg (1234) 

1 HumRRO 

Joseph A. Austin Building 

1939 Goldsmith Lane 

Lousville, KY 40218 (3) 

1 Dr. Lawrence B. Johnson 

Lawrence Johnson & Associates, Inc. 
2001 S Street. N.W., Suite 502 
Washington, D.C. 20009 (All) 

1 Dr. Arnold F. Kanarick 
Honeywell, Inc . 
2600 Ridge Parkway 
Minneapolis, MN 55413 (3) 

1 Dr. Roger A. Kaufman 

U.S. International University 

Graduate School of Human Behavior 

Elliott Campus 

8655 E. Pomerada Road 

San Diego, CA 92124 (134) 

1 Dr. Steven W. Keele 
University of Oregon 
Department of Psychology 
Eugene, OR 97403 (3) 

1 Dr. David Klahr 

Carnegie-Mellon University 
Department of Psychology 
Pittsburgh, PA 15213 (13) 

1 Dr. Alma E. Lantz 
University of Denver 
Denver Research Institute 
Industrial Economics Division (134) 



HumRRO Central Division 
Columbus Office 
Suite 23 

2601 Cross Country Drive 
Columbus, GA 31906 



(1234) 



Denver, CO 80210 

Mr . Brian McNally 
Educational Testing Service 
Princeton, NJ 08540 
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1 Dr. Robert R. Mackie 

Human Factors Research, Inc., 
6780 Corton Drive 
Santa Barbara Rp«earch Park 
Goleta, CA 93017 (13) 

1 Dr. William C. Mann 

University of Southern California 
» Information Sciences Institute 
4676 Admiralty Way 
Marina del Rey, CA 90291 (13) 

1 Dr. Leo Munday, Vice President 
American College Testing Program 
P.O, Box 168 
Iowa City, lA 52240 

(1234) 

1 Dr. Donald A. Norman 

University of California, San Diego 

Department of Psychology 

La Jolla, CA 92037 (3) 

1 Mr. A. J. Pesch, President 
Eclectech Associates, Inc. 
P.O. Box 178 

North Stonington, CT 06359 (13) 

1 



1 



1 



1 



1 Dr. George E. Rowland 
Rowland and Company, Inc. 
P.O. Box 61 

Haddonfleld, NJ 08033 

1 Dr. Arthur I. Siegel 

Applied Psychological Services 

404 East Lancaster Avenue 

Wayne, PA 19087 (1234) 

1 Dr. Richard Snow 
Stanford University 
School of Education 
Stanford, CA 94305 (3) 

1 Dr. C. Harold Stone 
1428 Virginia Avenue 
Glendale, CA 91202 

1 Mr, Dennis J. Sullivan 

c/o HAISC, Building 119, M,S. 2 
P.O. Box 90515 

Los Angeles, CA 90009 (123) 

1 Dr, Patrick Suppes 
Stanford University 
Institute for Mathematical Studies 



Mr. Luigl Petrullo 
2431 North Edgewood Street 
Arlington, VA 22207 



(AH) 



Dr, Steven M. Pine 
University of Minnesota 
Department of Psychology 
Minneapolis, MN 55455 



(123) 



Dr. Diane M. Ramsey-Klee 

R-K Research & System Design 

3947 Ridgemont Drive 

Malibu, CA 90265 " • (13) 

Dr. Joseph W. Rigney 
University of Southern California 
Behavioral Technology Laboratories 
3717 South Grand 

Los Angeles, CA 90007 (1234) 



Dr. Leonard L. Rosenbaum, Chairman 
Montgomery College 
Department of Psychology 
Rockville, MD 20850 (1234) 



in the Social Sciences 
Stanford, CA 94305 (3) 

Dr. K, W, Uncapher 

University of California 

Information Sciences Institute 

4676 Admiralty Way 

Marina del Rey, CA 90291 (13) 

Dr. Benton J. Underwood 
Northwestern University 
Department of Psychology 
Evanston, IL 60201 (3) 



Dr. Carl R. Vest 
Battelle Memorial Institute 
Washington Operations 
2030 M Street, N.W. 
Washington, D.C. 20036 



(34) 



Dr. David J. Weiss 
University of Minnesota 
Department of Psychology 
N660 Elliott Hall 
Minneapolis, MN 55455 



(123) 
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1 Dr. K. Wescourt 
Stanford University 
Institute for Mathematical Studies in 

the Social Sciences 
Stanford, CA 94305 

1 Dr. Anita West 
'/ Denver Research Institute 
University of Denver 
Denver, CO 80210 (13) 

1 Dr. Kenneth N. Wexler 
University of California 
School of Social Sciences 
Irvine, CA 92664 (3) 

1 Mr. E. A. Dover 

2711 South Veitch Street 

Arlington, VA 22206 (All) 
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